Address certificate loading regression #6731
Conversation
nateprewitt
force-pushed
the
consider_certs
branch
from
ca8df2d
to
e341df3
Compare
| pool_kwargs["ssl_context"] = _preloaded_ssl_context | ||
| elif verify is True: | ||
| # Set default ca cert location if none provided | ||
| cert_loc = extract_zipped_paths(DEFAULT_CA_BUNDLE_PATH) |
There was a problem hiding this comment.
So this is what fixes #6730 right? Because the custome Context + passing this should make their life better, right?
There was a problem hiding this comment.
Yeah, this should address #6730. #6667 assumed we would always be using the default context which lead to removing these two lines. That was "ok" because we were always forcing a load on the default context at import time.
Starting in 2.32.3, we've opted out of the default in the failure cases reported between 2.32.0-2.32.2. These are cases we're taking a custom context from the user but no longer setting the DEFAULT_CA_BUNDLE on the connection returned from what was previously get_connection. Since we don't know if we have the default SSLContext in cert_verify, I ported this into _urllib3_request_context to ensure we're passing this as pool_kwargs when we've disabled the default.
I validated this against the issue reported in #6726 which was showing this regression after the cert fix in 2769cb6. The other reported issues like #6730 are also passing cleanly now.
|
Is there an ETA for this change or a similar change? 2.32.3 doesn't work for us, as it broke our usage of |
…e#1583) Requests prior to 2.32.3 always loaded the default (system-wide) set of trusted certificates into custom SSL contexts. 2.32.3 no longer does. This has broken a lot of users, but the fix is moving slowly upstream due to security considerations - see psf/requests#6730 and psf/requests#6731 . As suggested at psf/requests#6710 (comment) this can be worked around by explicitly loading the default certificates into the context. We check the method exists before calling it just to be safe, but I'm pretty sure it's been there as long as this interface has existed. Signed-off-by: Adam Williamson <[email protected]>
…e#1583) Requests prior to 2.32.3 always loaded the default (system-wide) set of trusted certificates into custom SSL contexts. 2.32.3 no longer does. This has broken a lot of users, but the fix is moving slowly upstream due to security considerations - see psf/requests#6730 and psf/requests#6731 . As suggested at psf/requests#6710 (comment) this can be worked around by explicitly loading the default certificates into the context. We check the method exists before calling it just to be safe, but I'm pretty sure it's been there as long as this interface has existed. Signed-off-by: Adam Williamson <[email protected]>
|
The regression is there for some time and the PR is still in draft. Is there some way to move this forward? |
…e#1583) Requests prior to 2.32.3 always loaded the default (system-wide) set of trusted certificates into custom SSL contexts. 2.32.3 no longer does. This has broken a lot of users, but the fix is moving slowly upstream due to security considerations - see psf/requests#6730 and psf/requests#6731 . As suggested at psf/requests#6710 (comment) this can be worked around by explicitly loading the default certificates into the context. We check the method exists before calling it just to be safe, it was added in Python 3.4. Signed-off-by: Adam Williamson <[email protected]>
|
Maintainers, is there any update here? The latest version of HTTPie is mostly non-functional due to this issue, and we will need to drop it and migrate to other tools if this cannot be addressed. There does not seem to be any blocking feedback on this PR - can any clarity be provided about what is currently preventing a merge? Is additional contribution required from the community? |
|
A friendly reminder that this issue has been stagnant for 4 months already. |
|
Hi @sigmavirus24 you're listed as reviewer for this. Would you like to review this or assign someone else to do so? |
#1596) * Explicitly load default certificates when creating SSL context (#1583) Requests prior to 2.32.3 always loaded the default (system-wide) set of trusted certificates into custom SSL contexts. 2.32.3 no longer does. This has broken a lot of users, but the fix is moving slowly upstream due to security considerations - see psf/requests#6730 and psf/requests#6731 . As suggested at psf/requests#6710 (comment) this can be worked around by explicitly loading the default certificates into the context. We check the method exists before calling it just to be safe, it was added in Python 3.4. Signed-off-by: Adam Williamson <[email protected]> * Drop the upper bound on the requests dependency again As we can now work with requests 2.32.3+, we no longer need this pin. Signed-off-by: Adam Williamson <[email protected]> --------- Signed-off-by: Adam Williamson <[email protected]>
Overview
This PR is intended to address two distinct issues introduced with the default cert optimizations originally introduced in 2.32.0. While we continue to refine the settings considered when opting into our optimized context, we'll no longer use the new default if any custom cert values are supplied. This addresses the concurrency issues raised in #6726.
The second piece of this will be ensuring that when opting out of the default SSLContext, we're still supplying to the default CA Cert bundle correctly. This addresses the problems noticed in #6710 (comment) and #6730.
Considerations
We're now duplicating a decent chunk of the logic from cert_verify inside _urllib3_request_context but without our validation exceptions. That's a potential vector for behavioral shifts in the future. We could consolidate some of this behavior in one place but it's going to require constructing a dict and using
setattron ourconnincert_verifywhile settingpool_kwargsin_urllib3_request_context. I started writing that up but it feels clunky. This is probably going to be a tradeoff of risking drift like we have with Session settings and binding the two behaviors together too tightly.Testing
I'd like to codify the issues we've encountered through the whole 2.32.x saga in tests to hopefully avoid this in the future. Doing it cleanly without relying on external endpoints is proving to be a bit more involved than I'd like. I think we can harness some of the infrastructure added in #6662, but I haven't had a chance to really dig into that.